WIP: Source Links Fix by MaximilianSoerenPollak · Pull Request #440 · eclipse-score/docs-as-code

MaximilianSoerenPollak · 2026-03-11T21:07:55Z

Still in progress, do not merge.
It works but need to fix tests and clean it up a lot.

Still TODO:

Achieve same or better test coverage as currently (see last comment)
- Adapt all current tests to work with new structure
- Add new tests to enable testing of new functionality
Update Documentation if needed
Clean up implementation (naming / debug prints etc. )
Proof it works in ref-integration & locally (docs-as-code & other local repos)

📌 Description

🚨 Impact Analysis

This change does not violate any tool requirements and is covered by existing tool requirements
This change does not violate any design decisions
Otherwise I have created a ticket for new tool qualification

✅ Checklist

Added/updated documentation for new or changed features
Added/updated tests to cover the changes
Followed project coding standards and guidelines

github-actions · 2026-03-11T21:08:33Z

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run --lockfile_mode=error //src:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.3.0) and connecting to it...
INFO: Invocation ID: a634a019-05f7-407a-9da0-e3ec9c0f5f65
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
    currently loading: src
Loading: 0 packages loaded
    currently loading: src
Analyzing: target //src:license-check (1 packages loaded)
Analyzing: target //src:license-check (1 packages loaded, 0 targets configured)
Analyzing: target //src:license-check (1 packages loaded, 0 targets configured)

Analyzing: target //src:license-check (66 packages loaded, 9 targets configured)

Analyzing: target //src:license-check (70 packages loaded, 9 targets configured)

Analyzing: target //src:license-check (76 packages loaded, 9 targets configured)

Analyzing: target //src:license-check (128 packages loaded, 1003 targets configured)

Analyzing: target //src:license-check (131 packages loaded, 2515 targets configured)

Analyzing: target //src:license-check (140 packages loaded, 2576 targets configured)

Analyzing: target //src:license-check (140 packages loaded, 2576 targets configured)

Analyzing: target //src:license-check (140 packages loaded, 2576 targets configured)

Analyzing: target //src:license-check (144 packages loaded, 4589 targets configured)

INFO: Analyzed target //src:license-check (145 packages loaded, 4715 targets configured).
[11 / 16] [Prepa] JavaToolchainCompileClasses external/rules_java+/toolchains/platformclasspath_classes
[13 / 16] JavaToolchainCompileClasses external/rules_java+/toolchains/platformclasspath_classes; 0s disk-cache, processwrapper-sandbox
[14 / 16] JavaToolchainCompileBootClasspath external/rules_java+/toolchains/platformclasspath.jar; 0s disk-cache, processwrapper-sandbox
INFO: Found 1 target...
Target //src:license.check.license_check up-to-date:
  bazel-bin/src/license.check.license_check
  bazel-bin/src/license.check.license_check.jar
INFO: Elapsed time: 23.276s, Critical Path: 2.76s
INFO: 16 processes: 12 internal, 3 processwrapper-sandbox, 1 worker.
INFO: Build completed successfully, 16 total actions
INFO: Running command line: bazel-bin/src/license.check.license_check src/formatted.txt <args omitted>
usage: org.eclipse.dash.licenses.cli.Main [-batch <int>] [-cd <url>]
       [-confidence <int>] [-ef <url>] [-excludeSources <sources>] [-help] [-lic
       <url>] [-project <shortname>] [-repo <url>] [-review] [-summary <file>]
       [-timeout <seconds>] [-token <token>]

github-actions · 2026-03-11T21:13:18Z

The created documentation from the pull request is available at: docu-html

src/extensions/score_source_code_linker/metadata.py

Copilot

Pull request overview

This PR reworks how “source code links” are generated and injected into Sphinx needs, aiming to support both local builds (derive Git hash from the current repo) and combo/ref-integration builds (use per-module metadata such as repo URL + commit hash).

Changes:

Add metadata support (module name / repo URL / commit hash) to NeedLink JSON handling and Bazel CLI generation/merge scripts.
Introduce grouping of needs by module and a new module-grouped cache JSON.
Update Sphinx extension flow to read the new caches and generate GitHub links using either git-derived or metadata-derived refs.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
src/helper_lib/additional_functions.py	Changes GitHub link generation API to require module metadata.
src/extensions/score_source_code_linker/needlinks.py	Extends NeedLink with metadata fields; adds new metadata-aware JSON format/load path.
src/extensions/score_source_code_linker/need_source_links.py	Moves `group_by_need` into this module.
src/extensions/score_source_code_linker/module_source_links.py	New module-grouped cache format and grouping logic.
src/extensions/score_source_code_linker/metadata.py	New TypedDict + TypeGuard for metadata records.
src/extensions/score_source_code_linker/generate_source_code_links_json.py	Refactors extraction helper signature and logging (currently inconsistent call sites).
src/extensions/score_source_code_linker/init.py	Adds module-linker stage and changes injection to use module-grouped cache + metadata-based link generation.
scripts_bazel/merge_sourcelinks.py	Merges per-module sourcelinks and enriches with known-good repo/hash metadata.
scripts_bazel/generate_sourcelinks_cli.py	Emits sourcelinks JSON with a leading metadata dict and uses updated extraction helper signature.
docs.bzl	Adds optional `known_good` wiring into the sourcelinks merge genrule and the public `docs()` macro.

You can also share your feedback on Copilot code review. Take the survey.

scripts_bazel/merge_sourcelinks.py

scripts_bazel/generate_sourcelinks_cli.py

Copilot · 2026-03-12T00:15:37Z

scripts_bazel/generate_sourcelinks_cli.py

+    store_source_code_links_with_metadata_json(
+        file=args.output, metadata=metadata, needlist=all_need_references
+    )


This switches the generated JSON format from a plain list of NeedLinks to a list whose first element is a metadata dict. Any existing consumers/tests that expect the old schema will now fail. Consider either keeping the old format as default (with an opt-in flag for metadata), or updating all in-repo consumers and tests in the same PR to avoid a partially-migrated state.

@a-zw Valid issue here .

Do you think it would be better to rename this specific cache that comes from here a bit so it is clear it is with metadata?
Like 'scl_metadata_cache.json' or whatever? SO that the name makes it clear to use the metadata reader?

src/extensions/score_source_code_linker/__init__.py

Copilot · 2026-03-12T00:15:39Z

src/helper_lib/additional_functions.py

+    metadata: moduleInfo,
+    link: NeedLink | DataForTestLink | DataOfTestCase | None = None,
+) -> str:
+    if link is None:
+        link = DefaultNeedLink()
+    if not metadata.hash:
+        # Local path (//:docs)
+        return get_github_link_from_git(link)
+    # Ref-Integration path (//:docs_combo..)
+    return get_github_link_from_json(metadata, link)


get_github_link now requires a metadata argument, but there are existing call sites in the repo that still call get_github_link(link) (e.g. in tests and xml_parser.py). This is a breaking API change that will raise TypeError. Consider keeping backwards compatibility by making metadata optional with a sensible default (e.g. infer local/git behavior when metadata is omitted) or providing a new function name for the metadata-based behavior.

Suggested change

metadata: moduleInfo,

link: NeedLink | DataForTestLink | DataOfTestCase | None = None,

) -> str:

if link is None:

link = DefaultNeedLink()

if not metadata.hash:

# Local path (//:docs)

return get_github_link_from_git(link)

# Ref-Integration path (//:docs_combo..)

return get_github_link_from_json(metadata, link)

metadata: moduleInfo

| NeedLink

| DataForTestLink

| DataOfTestCase

| None = None,

link: NeedLink | DataForTestLink | DataOfTestCase | None = None,

) -> str:

"""

Return a GitHub link for the given location.

Backwards compatible calling conventions:

- New style: get_github_link(metadata, link)

- Old style: get_github_link(link) # metadata omitted

"""

# Distinguish between new-style and old-style calls.

if isinstance(metadata, moduleInfo):

actual_metadata: moduleInfo | None = metadata

actual_link = link

else:

# Old-style: first argument is actually the link (or None)

actual_metadata = None

# If both metadata and link are provided but metadata is not a moduleInfo,

# prefer the explicitly provided link argument.

actual_link = link if link is not None else metadata

if actual_link is None:

actual_link = DefaultNeedLink()

# If no metadata is available or metadata.hash is falsy, fall back to git-based link.

if actual_metadata is None or not getattr(actual_metadata, "hash", None):

# Local path (//:docs)

return get_github_link_from_git(actual_link)

# Ref-Integration path (//:docs_combo..)

return get_github_link_from_json(actual_metadata, actual_link)

src/extensions/score_source_code_linker/need_source_links.py

src/extensions/score_source_code_linker/needlinks.py

src/extensions/score_source_code_linker/module_source_links.py

src/extensions/score_source_code_linker/needlinks.py

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 8 comments.

You can also share your feedback on Copilot code review. Take the survey.

src/extensions/score_source_code_linker/helpers.py

src/extensions/score_source_code_linker/generate_source_code_links_json.py

src/extensions/score_source_code_linker/module_source_links.py

src/extensions/score_source_code_linker/__init__.py

scripts_bazel/merge_sourcelinks.py

docs.bzl

src/extensions/score_source_code_linker/need_source_links.py

src/extensions/score_source_code_linker/module_source_links.py

MaximilianSoerenPollak · 2026-03-12T15:54:48Z

Just realized that testlinks also need the module name earlier than source_code_linker extension end state.
As well as the edgecase that it could be that source_code_links are empty but testlinks exists. Then we have 0 idea about the module state.

I need to re-think this approach a bit and see what I can adapt to make this more convenient. I think I have an idea, but unsure.

MaximilianSoerenPollak · 2026-03-13T15:54:20Z

Still a bit to do here, but the architecture is 'working', though not pretty.
Will work on fixing the tests soon, and fixing the finaly bug in the testlinks that the actuall github link is put together wrong

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 14 comments.

You can also share your feedback on Copilot code review. Take the survey.

src/extensions/score_source_code_linker/generate_source_code_links_json.py

    # Use os.walk to have better control over directory traversal
    for file in iterate_files_recursively(search_path):
-        references = _extract_references_from_file(search_path, file)
+        references = _extract_references_from_file(search_path,Path(file.name), file)


src/extensions/score_source_code_linker/helpers.py

src/extensions/score_source_code_linker/needlinks.py

MaximilianSoerenPollak · 2026-03-16T10:17:11Z

src/extensions/score_source_code_linker/generate_source_code_links_json.py

    assert root.is_absolute(), "Root path must be absolute"
-    assert not file_path.is_absolute(), "File path must be relative to the root"
+    assert not file_path_name.is_absolute(), "File path must be relative to the root"
    # assert file_path.is_relative_to(root), (
    #     f"File path ({file_path}) must be relative to the root ({root})"
    # )
-    assert (root / file_path).exists(), (
-        f"File {file_path} does not exist in root {root}."
+    assert (root / file_path_name).exists(), (
+        f"File {file_path_name} does not exist in root {root}."
    )


This has NOT failed so far, this 'fidning' is wrong

src/extensions/score_source_code_linker/module_source_links.py

src/extensions/score_source_code_linker/BUILD

src/extensions/score_source_code_linker/helpers.py

scripts_bazel/merge_sourcelinks.py

src/extensions/score_source_code_linker/need_source_links.py

MaximilianSoerenPollak · 2026-03-16T10:40:46Z

src/extensions/score_source_code_linker/xml_parser.py

+    if module_name != "local_module" and known_good_json:
+        md["hash"], md["url"] = parse_info_from_known_good(
+            Path(known_good_json), module_name
+        )


Getting fixed

MaximilianSoerenPollak · 2026-03-16T12:56:13Z

Current TestCoverage report of main:

Goal => Do not fall under current coverage after changes.

After just converting the current tests to new structure this is the new test coverage:

MaximilianSoerenPollak added 2 commits March 11, 2026 20:48

WIP: Local Works somewhat

7dece30

WIP: seems to work in ref & local

ebdb40f

github-project-automation bot added this to INF - Infrastructure Community Mar 11, 2026

github-project-automation bot moved this to Backlog in INF - Infrastructure Community Mar 11, 2026

MaximilianSoerenPollak commented Mar 11, 2026

View reviewed changes

src/extensions/score_source_code_linker/metadata.py Outdated Show resolved Hide resolved

MaximilianSoerenPollak mentioned this pull request Mar 11, 2026

docs testlinks should point to version from bazel_dep #412

Open

MaximilianSoerenPollak requested a review from Copilot March 12, 2026 00:08

Copilot started reviewing on behalf of MaximilianSoerenPollak March 12, 2026 00:08 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

MaximilianSoerenPollak added 7 commits March 12, 2026 13:24

Remove metadata file

7f9f8bb

Formatting & Linting

403b9e9

Known_good required in merge script

bb73a0f

Remove debug print statements

703cc71

Copilot Findings

59f7602

Formatting & Linting

7155441

Basepyright linting

2355aa3

MaximilianSoerenPollak requested a review from Copilot March 12, 2026 13:26

Copilot started reviewing on behalf of MaximilianSoerenPollak March 12, 2026 13:26 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

MaximilianSoerenPollak added 2 commits March 13, 2026 16:51

WIP: Testlinks endlink still wrong

bb475e8

Formatting

996a51e

Formating & Linting & Comments

9a9ebab

MaximilianSoerenPollak requested a review from Copilot March 16, 2026 10:01

Copilot started reviewing on behalf of MaximilianSoerenPollak March 16, 2026 10:01 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

Fixing copilot & other findings

0edad55

-    metadata: moduleInfo,
-    link: NeedLink | DataForTestLink | DataOfTestCase | None = None,
-) -> str:
-    if link is None:
-        link = DefaultNeedLink()
-    if not metadata.hash:
-        # Local path (//:docs)
-        return get_github_link_from_git(link)
-    # Ref-Integration path (//:docs_combo..)
-    return get_github_link_from_json(metadata, link)
+    metadata: moduleInfo
+    | NeedLink
+    | DataForTestLink
+    | DataOfTestCase
+    | None = None,
+    link: NeedLink | DataForTestLink | DataOfTestCase | None = None,
+) -> str:
+    """
+    Return a GitHub link for the given location.
+    Backwards compatible calling conventions:
+    - New style: get_github_link(metadata, link)
+    - Old style: get_github_link(link)  # metadata omitted
+    """
+    # Distinguish between new-style and old-style calls.
+    if isinstance(metadata, moduleInfo):
+        actual_metadata: moduleInfo | None = metadata
+        actual_link = link
+    else:
+        # Old-style: first argument is actually the link (or None)
+        actual_metadata = None
+        # If both metadata and link are provided but metadata is not a moduleInfo,
+        # prefer the explicitly provided link argument.
+        actual_link = link if link is not None else metadata
+    if actual_link is None:
+        actual_link = DefaultNeedLink()
+    # If no metadata is available or metadata.hash is falsy, fall back to git-based link.
+    if actual_metadata is None or not getattr(actual_metadata, "hash", None):
+        # Local path (//:docs)
+        return get_github_link_from_git(actual_link)
+    # Ref-Integration path (//:docs_combo..)
+    return get_github_link_from_json(actual_metadata, actual_link)

Conversation

MaximilianSoerenPollak commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📌 Description

🚨 Impact Analysis

✅ Checklist

Uh oh!

github-actions bot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License Check Results

Uh oh!

github-actions bot commented Mar 11, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

MaximilianSoerenPollak Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MaximilianSoerenPollak commented Mar 12, 2026

Uh oh!

MaximilianSoerenPollak commented Mar 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

MaximilianSoerenPollak Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MaximilianSoerenPollak Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

MaximilianSoerenPollak commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MaximilianSoerenPollak commented Mar 11, 2026 •

edited

Loading

github-actions bot commented Mar 11, 2026 •

edited

Loading

MaximilianSoerenPollak commented Mar 16, 2026 •

edited

Loading